Repository navigation
UCT/ZE/ZE_IPC: enable level zero ipc support for Intel GPUs - #11218
Conversation
|
@zhangxiaoli73 Please fix the commit titles and do a rebase to resolve the conflicts. Maybe consolidate the two commits at the same time since a forced push is inevitable. @yosefe I am assuming forced push is allow in this case. Is that right? If not, what is the recommended way to fix the commit titles? |
Got it. Let me know if I can force push to change the commit. |
118b1f6 to
8b1525f
Compare
|
@yosefe Please help to review. |
|
@zhangxiaoli73 @yafshar can we add a build with ZE to builds.sh so it will run in CI with at least basic compilation check? |
Yes, we can add a ZE compile check to builds.sh and run it in CI. For this to be a meaningful ZE check, the selected Linux job must have Level Zero headers and the ze_loader development package available (or an equivalent module/path setup). We should also force configuration with --with-ze so the job fails when ZE dependencies are missing. If we rely on the default auto-detect mode, ZE may be silently skipped and the build could pass without actually validating ZE compilation.
In parallel, I will investigate a production-grade ZE mock/simulation option and follow up with another separate PR, as that work is broader in scope. |
|
hi @yafshar , |
8f6d46f to
5b3f330
Compare
| set -eEx | ||
| if [ "${install_ze_deps}" != "yes" ]; then | ||
| exit 0 | ||
| fi | ||
|
|
||
| # The Azure container runs as a non-root user; use sudo when | ||
| # available to install the Level Zero development headers. | ||
| SUDO= | ||
| if [ "$(id -u)" -ne 0 ]; then | ||
| SUDO="sudo -E" | ||
| fi | ||
|
|
||
| export DEBIAN_FRONTEND=noninteractive | ||
| $SUDO apt-get update | ||
| $SUDO apt-get install -y --no-install-recommends libze-dev || \ | ||
| $SUDO apt-get install -y --no-install-recommends level-zero-dev | ||
| displayName: Install ZE dev package | ||
| env: | ||
| install_ze_deps: $(install_ze_deps) |
There was a problem hiding this comment.
is there a publicly accessible container with ZE libs installed? it will be more efficient (less code, shorter time) than installing in each CI
There was a problem hiding this comment.
Done.
The public intel/deep-learning-essentials image is missing autoconf/automake/libtool/m4/doxygen, and Azure forces container jobs to run as non-root so we cannot apt-get install them at runtime. I wrapped it in a small builder image (recipe added at buildlib/dockers/ze-builder.Dockerfile), same pattern as the CUDA/ROCm lanes.
The image is temporarily published to my personal GHCR (ghcr.io/yuanwu2017/ucx-ze-builder:ubuntu24.04) so CI can come up. Could you rebuild this Dockerfile and push it to MLNX Harbor?
| ze) | ||
| tests=('build_ze') | ||
| ;; |
There was a problem hiding this comment.
why need special 'ze' mode, if we already add it to base_tests?
can use test_mode=short - same as done for ROCm container
There was a problem hiding this comment.
You are right. Removed it.
Fixed. |
|
@yosefe Please help to review and merge the patch. |
|
@yosefe Please help to review. |
|
@yosefe Bad commit title: 'Change local/remote handle names (#11145)' |
| Michal Shalev <mshalev@nvidia.com> | ||
| Mike Dubman <miked@mellanox.com> | ||
| Mikhail Brinskii <mikhailb@nvidia.com> | ||
| Mikhail Brinskiy <brminich@users.noreply.github.com> |
There was a problem hiding this comment.
Removing those lines breaks the AUTHORS file check CI job: contrib/authors_update.sh.
|
Rebased on latest
No changes to the feature code under Local test on real Intel GPU ( Configure: All ZE gtests pass:
|
Open the IPC handle on the iface device, whose command lists run the
copy. The rkey carried the exporter's device ordinal in dev_num and the
importer used it to pick a local device, which names the wrong GPU
whenever the two processes enumerate GPUs in different orders. Remove
dev_num from the key together with uct_ze_base_get_device_ordinal. That
helper matched only root-device handles, so registering memory allocated
on a sub-device failed with UCS_ERR_INVALID_ADDR.
When the shared event pool is exhausted, give the operation a private
one-event pool rather than failing with UCS_ERR_NO_RESOURCE.
ep_pending_add cannot queue the operation, so UCP would retry the send
in a tight loop. The shared pool size is now configurable with
UCX_ZE_IPC_EVENT_POOL_SIZE (default 1024).
Stop advertising EVENT_SEND_COMP and EVENT_RECV. The iface has no event
fd and its event_arm returns UCS_ERR_UNSUPPORTED, which
ucp_worker_iface_check_events_do treats as fatal ("failed to arm
iface"); ucx_perftest aborted that way at teardown.
Duplicate the fd from a remote IPC handle whenever it is non-negative
and reject a negative one. The old (0, 65536) bound skipped the dup for
fd 0 and for fds of 65536 and above.
Remove the IPC handle dump and checksum helpers and the hand-rolled
timeline profiling. Drop the per-registration and per-query info logs
and move the iface init logs to debug. Report the actual cause in the
ze_copy missing-device error and indent Makefile.am with spaces. Rename
the get_device_by_ordinal_round_trip test and drop the tests of the
removed helper.
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
bb11f63 to
0156ae4
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The AddressSanitizer/roce worker-3 job did not fail on any test — all 8707 gtests passed — it was killed because the self-hosted Azure agent on Full analysisSummary: Root cause: Agent-side infrastructure fault, not a test or product failure. Evidence from the fetched log:
Implicated commit: none — the failure is unrelated to File: no product file implicated. Relevant CI config: Suggested fix:
Related: none found — searches for the agent-shutdown/cancellation signature returned only unrelated PRs (#11932, #11842, #11948, #11866, #11834), no existing issue tracking this flake.
|
brminich
left a comment
There was a problem hiding this comment.
pls add a cross-process ZE IPC RMA test, or point to existing coverage for that path. The new RMA tests use two endpoints in one process, so they bypass pidfd_getfd, which the intended interprocess use requires.
|
@brminich thank you for the catch, and agreed ... the current UCT RMA tests use the same-PID path, so they do not exercise pidfd_getfd. We will add a real exporter/importer test with an exec'ed peer, verify PUT/GET data and cache/FD cleanup on Intel hardware, and then report here. |
The existing ze_ipc RMA tests keep both entities in one process, so the transport takes its same-pid path and never exercises pidfd_getfd() or the cross-process reachability check. Fork and exec the gtest binary as an importer that unpacks the exporter's rkey and runs put_zcopy and get_zcopy against the exporter's buffer for several rounds, with and without the IPC cache. The exporter checks each put and fills the buffer for the following get. Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
While a zeMemGetIpcHandle() handle is held, the Level Zero driver (seen on libze-intel-gpu1 26.27.39122.11) closes the handle's dmabuf fd on the first command-list operation that a device other than the allocation's device issues on the buffer, and a later zeMemPutIpcHandle() closes the same fd number again. ze_ipc packed that fd number into the rkey and held the handle until dereg, so a peer's pidfd_getfd() failed with EBADF or duplicated an unrelated file that had reused the number, and the deferred put could close that file in our process. Duplicate the fd right after getting the handle, return the handle at once and pack the duplicate; dereg closes it. The duplicate keeps the dmabuf alive for importers, and no driver handle stays outstanding for the driver to close under us. Add a cross-process test that fills the exported buffer from a second device before the importer opens it. It is skipped when fewer than two devices are visible. Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
| * Progress all active command list queues | ||
| * Similar to CUDA IPC's uct_cuda_base_progress_event_queue | ||
| */ | ||
| ucs_queue_for_each_extract(q_desc, &iface->active_queue, queue, 1) { |
There was a problem hiding this comment.
blocker: ucs_queue_for_each_extract pulls the head on every iteration, so pushing q_desc back at the end of the body makes active_queue non-empty again and the loop immediately re-pulls the same descriptor. As long as one event is ZE_RESULT_NOT_READY, count never advances and progress spins forever. pls iterate like uct_cuda_base_iface_progress() does — ucs_queue_for_each_safe plus ucs_queue_del_iter once the event queue drains.
Verified: ucs_queue_for_each_extract (src/ucs/datastruct/queue.h:262) has ucs_queue_pull_non_empty(queue) in its loop condition and re-reads (queue)->head each iteration; the max_poll guard at L277 only triggers after a completion increments count.
There was a problem hiding this comment.
Fixed in ad67476. Progress now iterates the active queue with ucs_queue_for_each_safe and removes a command list with ucs_queue_del_iter once it drains, like uct_cuda_base_iface_progress(). Within a list it completes events in posting order and stops at the first unsignaled one, so a progress call returns instead of spinning until every outstanding copy completes. Each call completes at most MAX_POLL events.
| .ep_put_zcopy = uct_ze_ipc_ep_put_zcopy, | ||
| .ep_pending_add = (uct_ep_pending_add_func_t)ucs_empty_function_return_busy, | ||
| .ep_pending_purge = (uct_ep_pending_purge_func_t)ucs_empty_function, | ||
| .ep_flush = uct_base_ep_flush, |
There was a problem hiding this comment.
blocker: copies are posted asynchronously on immediate command lists, but uct_base_ep_flush returns UCS_OK unconditionally, so a caller can observe a PUT/GET as complete while the copy is still in flight. cuda_ipc uses uct_cuda_base_ep_flush and rocm_ipc uses uct_rocm_ipc_ep_flush for exactly this reason — pls add a real ep_flush that waits on the outstanding events.
There was a problem hiding this comment.
Fixed in ad67476. uct_ze_ipc_ep_flush returns UCS_OK when nothing is outstanding. Otherwise it queues a marker behind the outstanding events on each active command list and returns UCS_INPROGRESS. Progress completes a marker when it reaches the head of its list, and the last marker invokes comp.
This follows uct_cuda_base_ep_flush, including two of its limits: the flush covers every command list on the iface, not just this endpoint's copies, and UCT_FLUSH_FLAG_CANCEL is not acted on. A flush still pending when the iface is closed is freed without invoking comp. Covered by test_ze_ipc_rma.ep_flush_waits_for_copies and ep_flush_pending_at_close.
| uct_ze_ipc_iface_t *iface = ucs_derived_of(tl_iface, uct_ze_ipc_iface_t); | ||
| unsigned i; | ||
|
|
||
| /* Check if all command list queues are empty */ |
There was a problem hiding this comment.
blocker: comp is ignored while this returns UCS_INPROGRESS, so the completion is never invoked and the caller waits forever.
| /* Check if all command list queues are empty */ | |
| if (comp != NULL) { | |
| return UCS_ERR_UNSUPPORTED; | |
| } | |
| /* Check if all command list queues are empty */ | |
| for (i = 0; i < iface->num_cmd_lists; i++) { |
uct_cuda_base_iface_flush() and uct_rocm_ipc_iface_flush() reject a non-NULL comp the same way.
There was a problem hiding this comment.
Fixed in ad67476. uct_ze_ipc_iface_flush returns UCS_ERR_UNSUPPORTED when given a completion, as uct_cuda_base_iface_flush does. Without one it returns UCS_INPROGRESS while any command list has outstanding events.
| .mem_attach = (uct_md_mem_attach_func_t) | ||
| ucs_empty_function_return_unsupported, | ||
| .detect_memory_type = (uct_md_detect_memory_type_func_t) | ||
| ucs_empty_function_return_unsupported, |
There was a problem hiding this comment.
mem_elem_pack and mem_elem_release are left NULL by this initializer, and uct_md_mem_elem_pack() / uct_md_mem_elem_release() dereference them unconditionally.
| ucs_empty_function_return_unsupported, | |
| .detect_memory_type = (uct_md_detect_memory_type_func_t) | |
| ucs_empty_function_return_unsupported, | |
| .mem_elem_pack = (uct_md_mem_elem_pack_func_t) | |
| ucs_empty_function_return_unsupported, | |
| .mem_elem_release = (uct_md_mem_elem_release_func_t) | |
| ucs_empty_function, |
Same as ze_copy_md.c.
There was a problem hiding this comment.
Fixed in ad67476. The md ops now set mem_elem_pack to ucs_empty_function_return_unsupported and mem_elem_release to ucs_empty_function.
|
|
||
| /* Store information for cache-based cleanup and event tracking */ | ||
| event_desc->dup_fd = local_fd; | ||
| event_desc->pid = ep->remote_pid; |
There was a problem hiding this comment.
uct_ze_ipc_map_memhandle() keys the remote cache on key->pid, but unmap (here and in iface_progress) passes event_desc->pid. pls store key->pid so map and unmap always resolve to the same cache.
| event_desc->pid = ep->remote_pid; | |
| event_desc->pid = key->pid; |
There was a problem hiding this comment.
Fixed in ad67476. The event desc stores key->pid, the same pid the map keyed the cache on, and every unmap path in ze_ipc_ep.c and in progress uses it.
| $(ZE_LIBS) | ||
| endif | ||
|
|
||
| if HAVE_ZE |
There was a problem hiding this comment.
pls merge this into the existing if HAVE_ZE block above: $(ZE_CPPFLAGS) and $(ZE_LIBS) are added twice, and $(ZE_LDFLAGS) belongs in gtest_LDFLAGS (as in the first block), not gtest_LDADD.
There was a problem hiding this comment.
Fixed in ad67476. There is now a single if HAVE_ZE block, which adds $(ZE_CPPFLAGS) and $(ZE_LIBS) once and puts $(ZE_LDFLAGS) in gtest_LDFLAGS.
|
|
||
| ucs_trace("remote addr %p rkey %p size %zu", (void*)remote_addr, | ||
| (void*)rkey, size); | ||
| ucs_trace("ze_copy_ep: zcopy %s remote_addr=%p local_buf=%p size=%zu", |
There was a problem hiding this comment.
minor: docs/LoggingStyle.md asks for lowercase messages and no = in values, e.g. "zcopy put remote_addr %p local_buf %p size %zu". Same for the new PUT_SHORT/GET_SHORT traces below.
There was a problem hiding this comment.
Fixed in ad67476. The zcopy, put_short and get_short traces are lowercase and use name value without =, per docs/LoggingStyle.md.
|
|
||
| typedef struct uct_ze_ipc_cache_hash_key { | ||
| pid_t pid; | ||
| uint32_t _pad; /* Ensure 8-byte alignment for ze_context */ |
There was a problem hiding this comment.
In src/uct/ze/ze_ipc/ze_ipc_cache.h — minor: _pad and UCS_S_PACKED contradict each other — packing removes the alignment the pad exists for. uct_ze_ipc_cache_hash_equal() compares fields rather than using memcmp, so both can go.
There was a problem hiding this comment.
Fixed in ad67476. Dropped both _pad and UCS_S_PACKED from the cache key.
|
Residual coverage gap: No test exercises the progress loop with an outstanding (not-yet-complete) event on a second command list, which is where the Residual coverage gap: |
|
🤖 CI Triage Agent — TL;DR: The ASan GPU gtest aborted inside Full analysisSummary: Root cause: The test iterates over every SL (0.. The 14 s gap is exactly the RC retry/timeout budget (progress was being made, not a hang, and no wall-clock kill occurred). Because the entities created by Implicated commit: unknown (not a code regression from 2188836; the test loop predates it — relevant test file history: File: test/gtest/uct/ib/test_ib.cc:728-742 (SL loop in Suggested fix:
Related: #10197 (GTEST/IB: Don't fatal if an SL is blocked), #10262 (UCT/IB/TEST: Configure available SLs in gtest — added |
uct_base_ep_flush returns UCS_OK while copies are still in flight, so UCP could complete a flush before the data had landed. ep_flush now queues a marker behind the events of each active command list, as the cuda transports do, and invokes the caller's completion once every marker has reached the head of its queue. Closing the iface with a flush still pending frees the flush without completing it. Progress now completes each command list's events in posting order, so a marker completes only after the copies posted before it. It stops at the first event that has not signaled instead of extracting the list and pushing it back. An event leaves its queue only after its completion has run, so a completion that posts on the same command list no longer adds the list to the active queue a second time. iface_flush rejects a completion, like the cuda transports. Also from review: unmap uses the rkey's pid, matching the map; the md defines mem_elem_pack and mem_elem_release; the cache hash key drops its manual pad and packing; ze_copy traces follow the logging style; and the gtest Makefile has a single HAVE_ZE block. Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
compute-runtime 26.27 keeps the fd that zeMemGetIpcHandle returned inside the opaque IPC handle. zeMemOpenIpcHandle treats a handle whose fd field equals that stored fd as unmodified, and fetches that fd number from the exporter with pidfd_getfd instead of using the fd it was given. The exporter packs a dup and puts the handle right away, which closes the original fd. So when the importer's pidfd_getfd copy lands on the same number, the driver fetches a closed fd, and the open fails with 0x78000004, or it fetches whatever file has since reused the number. The rkey now carries the exporter's original fd. When the importer's copy has the same number, it moves the copy to another fd. The new xproc test forces the collision by filling the importer's fd table up to the exporter's fd, and checks that the importer's copy was moved. Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
Replace hardcoded torch.cuda calls in the Encoder/Prefill/Decode path with torch.get_device_module equivalents, so staging-buffer KV transfer works on Intel XPU. Two of the fixes are not XPU-specific: the encoder /send synchronize was gated on is_cuda and so was skipped on every other accelerator, and staging init defaulted device_type to "cuda". Breaking change for out-of-tree backends: init_staging_buffers, init_staging_allocator and init_dcp_pack_buffers take a required keyword-only device_type. The "cuda" default was dropped so no backend can silently allocate staging on CUDA. The DCP pack kernel is Triton-only, so off CUDA init_dcp_pack_buffers returns no buffers and DCP transfers fall back to per-token RDMA. Adds create_device_event and current_device_stream to srt/utils/common.py. The XPU PD tests run every scenario on both the NIXL and Mooncake backends. The NIXL classes are skipped: UCX treats SYCL device allocations as host memory, so NIXL cannot register XPU KV buffers until ai-dynamo/nixl#1536 and openucx/ucx#11218 land. The Mooncake classes run in CI (nightly-xpu-2-gpu and nightly-xpu-4-gpu). docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF) and installs sglang-router. The Mooncake classes skip when mooncake.engine is not importable, so the nightly stays green on an image built before that Dockerfile step. Mooncake's XPU support lives in its TENT engine, so those classes set MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp and need no RDMA NIC. The staging decode server (TP 2) runs with TORCHDYNAMO_DISABLE=1: at TP > 1 the compiled vocab-parallel embedding segfaults in inductor's static Triton launcher on XPU. Verified on 3x Intel Arc Pro B60: xpu/test_disaggregation_staging_xpu.py (prefill TP 1, decode TP 2, so KV goes through the staging path), 4 passed; test/registered/unit/disaggregation, 487 passed. With Mooncake over TCP: disaggregation/test_disaggregation_xpu.py 2 passed, staging 4 passed. The same Mooncake tests pass against a transfer-engine-only install of upstream Mooncake 16b7ba3c built with the Dockerfile steps.
|
@zhangxiaoli73 can you pls confirm you have CLA signed? |
|
@brminich Yes, Intel has CLA signed. @zhangxiaoli73 is covered by that, so do @yuanwu2017 and @tsg- . |
Replace hardcoded torch.cuda calls in the Encoder/Prefill/Decode path with
torch.get_device_module equivalents, so staging-buffer KV transfer works on
Intel XPU. Two of the fixes are not XPU-specific: the encoder /send
synchronize was gated on is_cuda and so was skipped on every other
accelerator, and staging init hardcoded a "cuda:{gpu_id}" device string.
Breaking change for out-of-tree backends: init_staging_buffers,
init_staging_allocator and init_dcp_pack_buffers take a required keyword-only
device_type. The "cuda" default was dropped so no backend can silently
allocate staging on CUDA.
The DCP pack kernel is Triton-only, so off CUDA init_dcp_pack_buffers returns
no buffers and DCP transfers fall back to per-token RDMA.
Adds create_device_event and current_device_stream to srt/utils/common.py.
The decode-side scatter now waits on an event recorded on schedule_stream when
the room is registered, right after pre-alloc writes its req_to_token row,
instead of having no ordering edge. Waiting on the stream itself would also
queue each scatter behind any forward running there without overlap.
The XPU PD tests run every scenario on both the NIXL and Mooncake backends.
The NIXL classes are skipped: UCX treats SYCL device allocations as host memory,
so NIXL cannot register XPU KV buffers until ai-dynamo/nixl#1536 and
openucx/ucx#11218 land. The Mooncake classes run in CI (nightly-xpu-2-gpu and
nightly-xpu-4-gpu). docker/xpu.Dockerfile builds the Mooncake transfer engine
from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON
-DWITH_STORE=OFF) and installs sglang-router. The Mooncake classes skip when
mooncake.engine is not importable, so the nightly stays green on an image
built before that Dockerfile step.
Mooncake's XPU support lives in its TENT engine, so those classes set
MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp and need no RDMA NIC. The staging
decode server (TP 2) runs with TORCHDYNAMO_DISABLE=1: at TP > 1 the compiled
vocab-parallel embedding segfaults in inductor's static Triton launcher on XPU.
Verified on 3x Intel Arc Pro B60: xpu/test_disaggregation_staging_xpu.py
(prefill TP 1, decode TP 2, so KV goes through the staging path), 4 passed;
test/registered/unit/disaggregation, 487 passed. With Mooncake over TCP:
disaggregation/test_disaggregation_xpu.py 2 passed, staging 4 passed. The same
Mooncake tests pass against a transfer-engine-only install of upstream Mooncake
built with the Dockerfile steps.
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit) and installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp, since CI has no RDMA NIC; TCP is temporary until kvcache-ai/Mooncake#4379 enables ze_ipc. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: UCX's ze_ipc transport maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights, so NIXL transfers fail with EPERM in a default container. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1 and the temporary MOONCAKE_PROTOCOL=tcp, host-staged TCP transport, no Mooncake store so no HiCache L3, a full prefill/ decode example) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit). The pin includes kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to card through Level Zero IPC and no RDMA NIC is needed. The image also installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards (same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights. In a default container NIXL transfers fail with EPERM, and Mooncake stops copying KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit). The pin includes kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to card through Level Zero IPC and no RDMA NIC is needed. The image also installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards (same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights. In a default container NIXL transfers fail with EPERM, and Mooncake stops copying KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit). The pin includes kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to card through Level Zero IPC and no RDMA NIC is needed. The image also installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards (same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights. In a default container NIXL transfers fail with EPERM, and Mooncake stops copying KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit). The pin includes kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to card through Level Zero IPC and no RDMA NIC is needed. The image also installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards (same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights. In a default container NIXL transfers fail with EPERM, and Mooncake stops copying KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


What?
Enable level zero IPC support for Intel GPUs, this PR adds:
Why?
We want to provide IPC transport for users within a single node.